Bundle Method for Structured Output Learning

Discriminative methods for learning structured output classifiers have been gaining popularity in recent years due to their successful applications in fields like computer vision, natural language processing or bio-informatics. Learning of the structured output classifiers leads to solving a convex minimization problem which is not tractable by standard algorithms. A significant effort has been put to development of specialized solvers among which the Bundle Method for Risk Minimization (BMRM) [Teo et al., 2010] is one of the most successful. The BMRM is a simplified variant of bundle methods well known in the filed of non-smooth optimization. The simplicity of the BMRM is compensated by its reduced efficiency. In this paper, we propose several improvements of the BMRM which significantly speeds up its convergence. The improvements involve i) using the prox-term known from the original bundle methods, ii) starting optimization from a non-trivial initial solution and iii) using multiple cutting plane model to refine the risk approximation. Experiments on real-life data show that the improved BMRM converges significantly faster achieving speedup up to a factor of 10 compared to the original BMRM. The proposed method has become a part of the SHOGUN Machine Learning Toolbox [Sonnenburg et al., 2010].